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ABSTRACT 

This report summarizes the results of an evaluation 
of the thinking skills model/program developed at the Mid-continent 
Regional Educational Laboratory (McREL). Participating were 19 
primary, 32 upper elementary, 10 junior high, and 16 senior high 
school teachers at four sites varying in size and locale. Providing a 
framework for teaching a wide runge of thinking skills within content 
area classrooms, the thinking skills model includes three general 
categories of cognitive skills considered relevant to academic 
success: (1) learning-to-learn skills, (2) content thinking skills, 
and (3) basic reasoning skills. Altogether, 18 skills areas are 
included in the model. Each is hypothesized to affect different 
student behaviors and to be associated with a particular rationale 
for the hypothesised effect; each was assessed with different 
measures within either one-shot case study or pretest-posttest 
designs. Trained in four spaced sessions, the 77 participating 
teachers pilot-tested the thinking skills techniques on over 1,90C 
students. Data were obtained through teacher-made tests, interviews 
with students, and observations of students by teachers. Results are 
reported for each skill area of the model: attention control, goal 
setting, attitude change, self-evaluation, concept attainment, 
pattern recognition, synthesizing, proceduralizing, deep processing, 
use of memory frameworks, categorizing, extrapolation, analogical 
reasoning, evaluation of evidence, evaluation of value, elaboration, 
problem solving, and invention. References are included. (RH) 
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INTRODUCTION 



The purpose of this report is to summarize the results of the 
efforts to evaluate the thinking skills program developed at the 
Mid-continent Regional Educational Laboratory. The program as 
initially conceives i: meant to be a framework for teaching a wide 
range cf thinking 3kills within content area classrooms. That is, the 
model was develop, * be as broad as possible (encompass as many 
skills as possible/ to be implemented in a regular classroom 

setting as opposed a special course or courses in "thinking 
skills. " 

Given this overall Intention the first step in the process of 
developing the McREL program was to survey the literature on cognition, 
artificial intelligence, developmental psychology, information 
processing and many other cognate fields. The research and theory from 
these fields were then translated into instructional strategies. That 
is, for the major theories and research findings within each of the 
cognate areas an attempt was made to develop a set of instructional 
practices. For some of the theories and research findings from these 
diverse fields, classroom application was difficult. In such cases, 
the theory or research finding was dropped from the model. The 
remaining components were then field tasted for teacher acceptance. 
Teachers from various grade levels re iewed the strategies and offered 
suggestions as to how strategies might be changed to improve their 
classroom effectiveness. During this second level screening some 
instructional techniques were also dropped from the model as a result 
of the feedback from teachers. The remaining strategies were then fit 
into a conceptual framework — a unifying theory of cognition which 
housed all strategies. Again, this was considered a level of 
screening. Strategies which survived the first two levels of screening 
were dropped from the model if they did not fit into the conceptual 
framework. 

This report describes the field testing results foi those 
strategies included in the conceptual framework, below a brief 
description of that framework is provided. For a detailed description 
see Marzano (1985a, 1985b) or Marzano and Hutchins (1985). 

THE THINKING SKILLS MODEL 

A primary thesis of the McREL thinking skills model is that a 
majority of the cognitive skills considered relevant to academic 
success can be categorized into three distinct groups: 1) 
learning-tc-learn skills, 2) content thinking skills and 3) basic 
reasoning skills. We consider each below: 
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Learning-to-Learn Skills : 



The learning- to- learn ski 1 Ls are those which fac i lit ate learning 
of all types. The basic assumption underlying the learning-to-learn 
skills is that learning within a classroom setting is a function of 
generalized competencies ths'% are used in all learning situations 
not just those related to school, ~f students are taught or made aware 
of these generalized competencies, they can use them in any situation 

school related and non-school related . There are four general 
competencies in the learning-to-learn area: 1) attention control, 2) 
goal setting, 3) monitoring att itudes and 4) sel f-evaluat ion. 

Attention control involves an awareness on the part of students of 
when they are and are not attending to a task. Within cognitive 
psychology, two types of attention are commonly identified: automatic 
and voluntary (Luria, 1983). Automatic attention is basically 
reflexive in nature; voluntary attention occurs when an individual 
willfully shifts attention to or maintains attention on a specific 
stimulus. The intent of the thinking ill entitled attention control 
is to provide students with an awareness rf when they are and are not 
attending and a set of strategies cor shifting attention when they so 
desire. 

Goal setting is the skill of identifying an explicit outcome and 
planning activities to accomplish that outcome. Over forty years ago, 
Sears (1940) found that successful students tended to set explicit 
goals. More recently Brophy (1982) found that successful students 
tended to set goals that were chal lenging but reasonable . Bandura and 
Schunk (1981) found that classroom goals should be short term 
(proximal) rather than long term (distal) to be most effective. 

Once a goal is set an individual will generally consider some high 
level controlling attitudes relative to che goal. A high level 
controlling attitude can be described as a basic operating principle 
which governs behavior. These principles are so important to human 
behavior that some theorist have hypothesized the tence of a 
specific type of memory housing then called "exec.tive memory. 11 (e.g. 
Sternberg, 1984). Within the thinking skills program students are 
taught to identify and foster specific attitudes which facilitate the 
accomplishment of academic goals (e.g. commitment to precision and 
involvement ) . 

During the attainment of a goal , effect i^e learners commonly 
engage in self-evaluation techniques in which they identify what is 
working and what is not working relative to the goal. This can be done 
both format wely and suramat ively . That is, students can monitor the 
effectiveness of their activities while they engage in a task and after 
they have completed the task. 
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Content Thinking Skills 

The purpose of the content thinking skills is to provide students 
with a set of information processing strategies which facilitate the 
learning of academic content sometimes referred to as domain specific 
knowledge (Doyle, 1983). There are four content thinking skills within 
the thinking skills program: 1) concept attainment, 2) pattern 
recognition, 3) synthesizing and 4) procedural izat ion. 

Concept att ainment is the process of att aining the cultural ly 
accepted labels for a set of experiences. According to Klausmeier and 
Sipple (1980) a concept is the "socially accepted meaning of one or 
more words which express the concept. 11 For example, the word dog is 
the label society uses to represent the conceptualization of a set of 
four-legged animals with certain characteristics. Within the thinking 
skills program concept attainment refers to: 1) the development of an 
experiential base for the concept, 2) the recognition of the auditory 
label for the concept , 3) the recognit ion of the orthographic label for 
the concept and \) the development of an accurate description of the 
concept . 

Patterns are organizational structures which hold together large 
blocks of information. There are two levels of patterns taught within 
the thinking skills program - basic patterns and macro-patterns. Basic 
patterns are used to organize linguistic information from the size of a 
paragraph to a few pages, or even a chapter of a textbook. 
Macro-patterns are used to organize even larger blocks of linguistic 
information (e.g. an entire book). 

Synthesizing is the process of expressing information read or 
heard in a streamlined version referred to as a macro-structure 
(Kintuch and van Dijk, 1978). This generally involves such techniques 
as identifying summary statements, generating summary statements when 
none exist and subsuming specific concepts under more general 
concept s • 

Procedural izat ion is the proc .ss of: 1) identifying the relevant 
steps in a specific process, 2) rehearsing these steps with the intent 
of M smoo thing them out 11 (dropping steps that are inefficient and adding 
steps that are more efficient), 3) practicing the process until it 
reaches a level of automat icity. This sequence for procedure learning 
was first described by Fitts (1964) and later expanded by Anderson 
(1983). 

Basic Reasoning Skills 

Reasoning skills are those considered basic to many cognitive 
tasks. According to Anderson (1983) reasoning skills can be subdivided 
into three general classes: 1) those that footer storage and retrieval 
of information, 2) those that match on* <*-*t- of information with tne 
other and 3) those that drastically restructure old information or 
build new information. 

-3- 
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1 . Storage and Retrieva l 



There are two basic storage and retrieval skills: 1) deep 
processing and 2) memory frameworks. Deep processing is the use of 
imagery to enhance incoming information. According to Sheikh (1983), 
imagery involves the creation of mental pictures, sounds, tastes, 
smells, noises and emotions about information. Apparently, with 
practice, people can enhance uheir imagery skills (Bellezza, 1983). 

Memory frameworks create "slots" in which sequential information 
can be deposited using deep processing. Within this category are loci 
methods (Ross & Lawrence, 1968) and pegword methods (Lindsay & Norman, 
1977). 

2. Matching Skills 

Matching skills are those which enable an individual to identify 
how incoming informat ion is similar Ko and different from informat ion 
stored in long terra memory. There are five types of matching skills: 
1) categorization, 2) extrapolation, 3) analogical reasoning, 4) 
evaluation of logic and 5) evaluation of value. 

According to Mervis (1980), categorization is an essential skill 
because "by categorizing a person is able to render the unfamiliar 
familiar and because one is able to generalize about an object based on 
knowledge about its category, one is able to know more about the object 
than just what can be ascertained by looking at it" (p 279). 

Within the thinking skills program, categorization is practiced as 
an adjunct skill to concept development. That is, students are asked 
to categorize new vocabulary words by their relevant attributes. 

Extrapolation is the process of matching the pattern of 
information read or heard with that of information from a different 
subject area or a different context. According to Alston (1964) this 
type of thinking is similar to the basics of metaphor. 

Analog ical reasoning is one of the most commonly included element s 
within thinking skills instructional models. According to Alexander 
(1984) few intellectual skills are as pervasive or essential as the 
ability to reason analogically. Within the McREL program, analogical 
thinking is taught us ing an adapt at ion of the four step process 
described by Sternberg (1977). 

Evaluation of evidence is the process of matching the structure of 
information with some formalized system of logic. Within this model, 
tha logic system developed by Toulmin (1958); (Toulrain, Rieke & Janik, 
1977) is used as the logic criterion. 



Evaluation of value is the process of matching information to some 
internalized system and than analyzing the logic of that value system. 
Spiro (1980) has stated that this "att itudinal" characteristic of 
thinking is the central aspect of cognition because it allows one to 
see the information base from which judgments are made. 

3. Knowledge Building & Reorganization 

There are three knowledge b llding and restructuring skills within 
the program: 1) elaboration, 2) {.roblera solving and 3) invention. 

Elaboration refers to inferring information not explicitly 
stated. Various categories of inference have been proposed by teachers 
and theorists (e.g. Bruce & Schmidt, 1974; Warren, 1979). Within this 
model three types of inferences are used: 1) elaboration of 
characteristics, 2) elaboration of casuality, 3) elaboration of general 
background and 4) elaboration of author purpose. 

Problem-solving occurs when an individual must "fill-ir" missing 
information. This is at the core of all problems; a goal is desired 
and information necessary to accomplish the goal is missing. Within 
the McREL model problem solving heuristics have been developed for 
general, everyday problems, mathematical problems, science problems, 
problems of cons: raining conditions and problems of unusual contexts. 

Invention is the process of creating new information or new 
products. This involves conceiving an idea, planning for its 
development, creating a working model and then revising or fine tuning 
the model until it reaches a polished state. Within mainstream 
education this process is generally taught as composing in written 
form. Within the thinking skills program it is taught as a general 
heuristic for the creation of any product. 

THE TRAINING PROGRAM 

The eighteen skill areas described above are generally presented 
to teachers as a single intervention - a comprehensive program. The 
order of present at ion to teachers generally follow that described 
above. That is, teachers are first presented techniques for the 
learning-to-learn skills, then techniques for the content thinking 
skills and finally techniques for the basic reasoning skills . The 
training usually takes four days (1 day for the learning-to-learn 
skills, 1 day for the content thinking skills, and 2 days for the 
reasoning skills). Each day of the training is spaced 2 to 5 weeks 
from preceding and subsequent training days. During the time between 
trainings participating teachers collect data on the effectiveness of 
the various techniques. At the end of the four days of training 
participating teachers use the data to make informed decisions as to 
which techniques work well for them and which techniques do not work 
well. For the most part it was teacher collected data that was used 
for the evaluat ion. 
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DEPENDENT VARIABLES 



Although the eighteen thinking skills are presented as a single 
intervention, they are hypothesized to affect different student 
behaviors . That is , the eighteen components of the independent variable 
(the thinking skilLs program) were hypothesized to affect different 
dependent variables . 



Below the dependent measures for each of the eighteen areas ar? 
briefly described. 





Skill Area 


Hypothes ized 
Effect 


Dependent 
Measure 


1. 


At tent ion 
control 


-increased task 
engagement 


-engaged time 
-e ng ag eme nt r at e 
-teacher observation 


2. 


Goal setting 


-increase* success 
in academic goals 


-success rate 
-teacher observation 


3. 


Monitor ing 
attitudes 


-increased effort 
at t r ibuu ion rel at ive 
to goals 


-student interviews 
-teacher observation 


4. 


Sel f-evaluat ion 


-increased raetacognit ive 
aware nes s of effect ive 
and inef feet ive act ions 


-s t udent int erv iews 
-teacher observation 


5 . 


Concept 
attainment 


-understanding of basic 
meaning of target 
concepts ; recognit ion 
of phonemic and 
orthograph represent- 
ation of the concept 


-teacher made 

vocabulary and 

spelling tests 
-teacher observation 


6. 


Pattern 
recognition 


-increased abil i*"y 
to organize large 
blocks of information 


-teacher made tests 
-teacher observation 


7. 


Synthesizing 


-increased abil it y 
to express the macro- 
structure of informa- 
t ion 


-teacher observation 


8, 


Proceduralizing 


-increased ability 
to internalize complex 
procedures 


-teacher observation 


9. 


Deep processing 


-increased abil it y to 


-teacher observat ion 



elaborate on the 
imagery characterist ics 

of inforraat ion 
-increased ability to 

recall informat ion 



10. Memory framework -increased ability to -teacher made tests 

recall informat ion 
sequent ially 
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11 . 


Cat ezor iz ine 


— increased ahilitv tn 

organize concepts into 
seraant ic categories 
-increased ability to 
identify shared 
attributes among sets 


— t* pap hp »» 

-teacher 


observation 


12, 


Extrapolat ion 


-increased abi lity to 
recognize similarities 

between sets of 
inf ormat ion from 
different sources 


-teacher 
-teacher 


made tests 
observation 


1 1 


An p 1 o^iffll 

reasoning 


— • i nr» ,"o jcori l 1 1 f v ffl 
-L ll v- L cdocu au x. *. a. k jr vj 

solve analogy problems 


-teacher 


m^Hp fpqfq 

observation 


14. 


Evaluation of 
evidence 


-increased abil ity 
to determine whether 
a claim is supported 
or unsupported 


-teacher 
-teacher 


made tests 
observat ion 



15. Evaluation of -increased ability to -student interviews 

value identify the assumptions -teacher observation 



underlying personal 
values and the basis for 
those assumptions 

16. Elaboration -increased ability to -teacher observation 

identify unstated 
characterist ics , causes 
and intentions 

17. Problem solving -increased metacognitive -teacher observation 

awareness of processes -student interviews 
for solving specific 
problems 
-increased abi lity to 
s o lve s pec i f ic prob lem 
types 

18. Invention -increased ability to -teacher observation 

conceive of a product 
and develop it to a 
final format 



A brief description of the rationale for the hypothesized effects 
for each component of the program is presented below: 

Skill Area Rationale for Hypothesized Effect 

1. Attention -increased awareness of level of attention 
control and incidence of attention will provide 
more student control over task engagement 



-7- 

8 



2. 


Goal setting 


-increased specificity of academic goals will 
provide heightened awareness of goal related 
act ivity and non-goal related act ivit y . This 
will increase goal related activity which will 
increase goal accomplishment. 


3. 


Monitoring 


-increased awareness of negSkt ive at t itudes 
about academic tasks along with cognitive 
restructuring activities will increase an 
effort attribution about success at school 
work. 


4. 


Sel f-evaluat ion 


-increased attention to feedback as to the 
formative fuccess of a task will increase 
student ability to judge the effectiveness of 
actions . 


5. 


Concept 


-identification of imagery characteristics of a 
concept along with semantic attributes will 
strengthen understanding , recal 1 and 
recognition of concepts. 


6. 


Pattern 
recognit ion 


-increased awareness of linguist ic patterns 
will increase ability to organize and retrieve 
text based inf ormat ion. 


7. 


Synthesizing 


-increased awareness of linguist ic pat terns 
will strengthen ability to identify important 
elements of linguistic information. 


8. 


Procedural! zing 


-increased awareness of the stages of 
procedural learning will increase attention to 
the stages which will improve acquisition of 
complex processes. 


9. 


Deep processing 


-increased awareness of imagery characteristics 
of in format ion wi 11 increase access to impl ied 
information. 

-increased attention to imagery characteristics 
of information will increase the number of 
cues associated with the information. 


10. 


Memory 


-use of pegwood and loci mnemonic methods will 
improve serial retrieval of inf ormat ion. 


11. 


Categorizing 


-increased awareness of types of semantic 
features will increase awareness of the 
different ways concepts are similar and 
disseminat ion. 


12. 


Extrapolation 


-increased awareness of shared organizational 
patterns of information will increase 
awareness of the ways large blocks of 
information can be similar and dissimilar. 


13. 


Analogical 
reasoning 


-increased awareness of relat ionships commonly 
involved in analogy problems will increase 
ability to solve analogies. 


14. 


Evaluation of 
evidence 


-increased awareness of informal falacies and 
unsuppor table claims will increase ability to 
identify unsubstantiated claims. 
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15. 


Evaluat ion 
of value 


-increased aware ness of the char act erist ics of 
value judgments wi 11 increase abi I ity to 
ident ify the underlying bel iefs and 
assumptions which are the basis for personal 
value judgment , 


16. 


Elaboration 


-increased awareness of the structure of 
various types of informat ion will increase 
access to stored information. 


17. 


Problem solving 


-increased awarenes3 of essent ial elements of 
various problem types (e.g., givens, 
operations, goals) will increase the 
efficiency of problem solving. 


Id. 


invent ion 


-increased awareness of phases of invent ion 
will increase the effectiveness of the 
invention process. 




In addition to 


the elements described above which were assumed to 



generate the hypothesized effects, each skill included a specific 
procedure to be taught to students. For sxample, for the skill of 
attention central students were presented with a specific set of steps 
which were considered heuristic rather than algorithmic (general 
"rules of thumb to follow 11 rather than a rigid set of steps). Beyer 
(1984) has suggested that lack of explicit instruction in procedures 
is one of the major causes of student failure to understand and 
utilize thinking skills. 



SUBJECTS 



The subjects for the evaluation were drawn from four sites where 
the thinking skills program was pilot tested. Those sites are 
described below: 

Site #1 was a large suburban school district serving over 6000 
students of mixed socio-economic strata. Twelve teachers from this 
district participated in the study. They represented the following 
grade levels: 



Grade Level No of Teachers 

Primary 4 



Upper Elementary 

Junior High 

Senior High 



Site #2 wa9 a srpall rural scL">ol district serving about 700 
students. Twenty-four teachers from the district participated in the 
study . The grade levels represented are 1 is ted below: 



G rade Level 

Primary 

Upper Elementary 

Junior High 

Senior High 



No. of Teachers 
8 

9 

4 
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Site #3 was a large rural district serving about 800 students. 
Thirty-five teachers participated in the: study: 



Grade Level No. of Teachers 

Primary 5 

Upper Elementary 16 

Junior High 4 

Senior High 10 

Site #4 was a single elementary school. Here six teachers 
participated in the study: 



Grade Level No. of Teachers 

Primary 2^ 



Upper Element ?~ y 



Including all four sites, the evaluation study included the 
following distribution of teachers: 



Grade Level No. of Teacher? 



Primary 


4 


+ 


8 


+ 


5+2-19 


Upper Elementary 


3 


+ 


$ 


+ 


16 + 4-32 


Junior High 


2 


+ 


4 


+ 


4 =-10 


Senior High 


3 


+ 


3 


+ 


10 -16 



Total ^77 



The 77 teachers involved in the study pilot tested the thinking 
skills techniques on over 1900 students. 



DATA COLLECTION 



For sites 1-3 data were collected by participating teachers 
between training sessions. That is, after training session #1 
participating teachers collected data on the techniques presented 
during session #1; after session #2 teachers collected data on the 
techniques presented in session #2; etc. At site #4 the data were 
collected by the trainer. Site #4 utilized a different training 
format. Participating teachers first went through all four days of the 
training. The techniques were then pilot tested in an affer school 
program in which students voluntarily participated. The trainer was 
present at all sessions and utilized those sessions to collect types of 
data that could not be collected from participating teachers. 



INSTRUMENTATION 



Al 1 tests used to assess academic performance were teacher made . 
No reliability or validity data were collected on those instruments. 
Similarly, the teacher observation data were collected in anecdotal 
fashion. That is, for the observational data, teachers simply recorded 
their percep . ions of student changes in behavior . For the success rate 
and engagement rate data collected by the trainer a formal observation 
systems were utilized. The trainer also used a formal system of 
protocol analysis to analyze the information obtained from student 
interviews . 
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limitations 



The limitations of the present evaluation effort were many. One 
obvious limitation is that the effects of the various thinking skill 
techniques can not be considered independent since the training was 
presented in a sequential fashion. Even though the theory base 
underlying the various skill areas would indicate that their effects 
might be relatively independent, the evaluation desigu was such that 
the skills taught earlier in the sequence could mask the effects of 
skil Is taught later in the sequence or c # este an eff ec t not relat ed to 
the technique under investigation. 

For most of the skill areas the evaluation design was what* 
Campbell and Stanley (1963) refer to as a one-shot case study. In such 
designs the dependent variable is measured after the intervent ion • 
According to Campbell & Stanley (1963) such a design has the 
following threats to internal and external validity: 



For those skill areas for which pretest data could be collected, 
a pretest-post test design was used. According to Campbell and Stanley, 
such a design has the following threats to internal and external 
validity: 



Eeiow the results for each of the 18 skill areas are discussed. 
Since all teachers in the study did not collect data on all skill 
areas, the nunber oZ ceachcrs reporting varies from skill area to skill 
area. 

1. Attention Control 

The effect for attention control was measured using teacher 
anecdotal comments, engaged time and engagement rate. Teacher 
anecdotal comments were reported by 65 teachers. Of those 51 reported 
that students were more aware of when they were and were not engaged 
and exhibited more control over their attending behavior as a result of 
the attention control process. Fourteen teachers reported no 
not iceable change in student attend ing behaviors • 



-history 
-maturation 

-interaction of select ion and treatment 



-history 
-maturation 
-test ing 

-instrument at ion 

-interact ion of select ion ,maturat ion , et c . 
-interaction of testing and treatment 
-interaction of selection and treatment 



RESULTS 
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Engaged time and engagement race were collected by the 
investigator in a pre-post Cushion in 6 Classrooms. Engaged time is 
expressed as a proportion of cotal class time. Engagement rate 
is expressed as the ratio of time engaged over the time devoted to 
academic tasks. The results for engaged time and engagement rate are 
reported below: 

Engaged Time 



Pre-test Post-test 

Mean Mean Difference df T-value Sig 

.75 .73 .02 5 .13 NS 



ENGAGEMENT RATE 

're-test Post-test 

Mean Mean Difference df T-value Sig 

Tin TYi TTU 5 Ttt TUT 



T-test results showed no significant difference in engaged time 
but did show a significant difference in engagement race. This 
indicates that students did not experience more engaged time probably 
due to the increased amount of non-instructional time necessary to 
administer the technique. However, students did exhibit a significant 
increase in engagement rate while instruction was occurring. Thid 
indicates that as a result of the attention control process students do 
exhibit more control over their attending behavior and more frequently 
choose to attend to the academic tasks presented to them. 

2. Goal Setting 

Goal set t" * ng was evaluated us ing teacher anecdot al dat a and 
succ'33 rate data. Anecdotal data were collected by 61 teachers. Of 
those, 53 reported that having students set explicit short term goals 
increased student success as measured by responses to teacher questions 
and in-class assignments . Fifteen part ic ipat ing teachers reported pre- 
and post-test classroom exercise data. Scores were transformed to 
proportions and two-tailed t-tests for dependent groups run: 



Pre-test Post-test 

Mean Mean Difference df T-value Sig 

.81 .88 1 07 175 4.05 01 

Pre-post engagement rate dat" were collected on 6 teachers. 

The results for a two-tailed, t-test for dependent groups are: 

Pre-test Post -test 

Mean Mean Difference df T-value Sig 

.79 .91 7T2 5 "8.17 .UUi 
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Pre-test and post-test analyses for both classroom exercise and 
success rate data showed significant gains at the .001 level. This was 
interpreted as an indication that increased attention to and 
specificity of academic goals increases raetacognitive control of the 
goal attainment process • 



3 . Monitoring Attitudes 

Student attitude change was assessed using teaching observation 
and student interview data. Teacher observation data were reporced by 
36 teachers. Of those, only 7 reported an observable change in student 
behavior as a result of the at t itude monitoring techniques . However , 
29 teachers also reported that they experienced an increased ability to 
communicate with the more troublesome students about specific problems 
within the class. 

Self report data were collected from 7 students. Of those, 5 
reported that they could see why certain attitudes about ce^ain 
classes were "getting in their way." These findings appear consistent 
with much of the research on attitude change — namely that attitudes 
can change without any observable behavior change. This might explain 
why only seven teachers reported changes in student behavior but 29 
reported more open connnunicat ion with troublesome students. It might 
be that students attitudes actually had begun to change resulting in 
more open communication. However, this attitude change had not as yet 
been translated into behavior change. 



4. Self-evaluation 

The effect of the self-evaluation technique were assessed using 
teacher observation and student interview data. Anecdotal data were 
reported by 34 of ^he 77 participating teachers. Of those, 29 stated 
that the self-evaluation techniques began to foster a self-awareness on 
the part of students of efficient and inefficient activities relative 
to completing a task. The remaining five reported no noticeable effect 
On students. 

Interview data were collected from 7 students - all 7 of which 
reported that the sel f-evaluat * 3n techniques made them more aware of 
when they were and were not being efficient. 

These findings were interpreted as an indication that 
self-evaluation techniques increase raetacognitive control of task 
performance and consequent efficiency of task behavior. 



5 . Concept Attainment 

Teacher observation and teacher made test data were collected to 
measure the effects of the concept attainment strategies. Sixty-five 
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participating teachers reported on the concept attainment process. Of 
those, 54 stated that they saw a definite improvement in students' 
knowledge of concepts as a result of the strategies. 

S venteen teachers reported pre-post scores on teacher made 
vocabulary tests. Results from a two-tailed, t-test for dependent 
groups are reported below: 

Pre-test Post-test 

Mean Mean Difference df T-value Sig 

.83 .94 Al 145 10.17 .001 

Eleven teachers reported pre-post scores on teacher made spelling 
tests. Results from a two-tailed t-test for dependent groups were: 

Pre-test Post-test 

Mean Mean Difference df T-value Sig 

.74 .81 .08 97 11.12 .001 



Both the vocabulary and spelling tests showed significant 
increases in scores as a result of the concept development techniques. 

These results were primarily atcributed to the imagery aspects of 
the concept attainment process. This was evident in the ancedotal 
comments of the teachers. That is, many of thoae teachers who reported 
an effect for the concept attainment process emphasized the importance 
aad power of the imagery component. 

6. Pattern Recognition 

Data on pattern recognition were collected via teacher observation 
and teacher made tests. Sixty-one teachers provided anecdotal 
data. Fifty-four of those, indicated that the pattern recognition 
techniques significantly improved students' abilities to understand 
material they read. 

Twelve teachers reported pre-post test dat a on student 
comprehension. Results from a two-tailed t-test for dependent groups 
are reported below: 

Pre- test Post -test 

Me a a Mean D: tference df T-value Sig 

.74 JJ3 .09 168 15.51 .001 

Results indicated a significant increase in students' comprehension 
ability as a result of the pattern recognition training. 



-14- 

15 



Further analysis of teacher anecdotal data strongly suggested that 
this is due to the use of patterns as an organizational tool. That is, 
students used the patterns as organizational frameworks which they 
superimposed on the information they read. 



Teacher observation was the only type of data collected on the 
skill of synthesizing. Twenty-three teachers reported on this 
technique . Of those , 1 7 indicated significant increase in study 
ability to synthesize information as a result of the technique. Again 
the anecdotal comments indicated that a key component of the 
synthesizing process was th« use of patterns as a tool for 
organization. 

8. Procedural izing 

Due to its relatively late inclusion in the thinking skills model, 
only four teachers reported on the effects of the proceduralizing 
techniques. All four indicated that this was a very difficult skill to 
teach students. None saw any immediate change in student behavior but 
all felt that over time this skill would effect useful changes in 
student behavior. More specifically the participating teachers 
reported that the level ot specificity reached exceeded that usually 
reached for classroom procedures. They stated that an increased 
attendion to detail would increase the level of learning for classroom 
procedures . 

9. Deep Processing 

Evaluation data for the technique, deep processing, were collected 
via teacher observation. Fifty-nine teachers reported. Forty-seven of 
those indicated that the deep processing technique increased students 1 
abilities to elaborate on the imagery characteristics of information 
and to recall information once it had been deep processed. More 
specifically, they reported that the imagery aspects of daep processing 
made students aware of aspects of information for which they were 
previously unaware. 

10 . Memory Frameworks 

Teacher made tests were used to evaluate the use of memory 
frameworks . Twenty- five teachers reported pre- post results : 



7. S ynthesizing 



Pre-test 
Mean 



Post-test 
Mean 



Difference 



df 



T-value 



Sig 



.46 



.78 



.32 



161 



25.71 



.001 
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A significant increase in students' abilities to recall 
information sequentially is indicated from these findings. 

The^e findings are consistent with much of the research on memory 
forks — namely that memory frameworks, increase retrieval of 
iatorraat ion . However, the difference between these findings and 
previous ones is the use of pattern recognition as a vehicle for 
organizing inforraat ion prior to the use of memory frameworks . More 
specifically most previous research was conducted using fairly simple, 
concrete concepts. Here large blocks of information were stored in 
each slot of the memory frameworks. The blocks were organized using 
patterns • 

1 1 . Categorizing 

Teacher observation and teacher made test data were used to assess 
the categorizing technique. Seventeen teachers provided anecdotal 
comments • Fourteen indicated that categorizing helped students 
understand the meaning of concepts and their defining attributes 

Six teachers turned in pre-post test data. The results of a 
two-tailed, t-test for dependent groups were: 

Pre- test Post- test 

Mean Mean Difference df T-value Sig 

.85 .88 1 03 45 1.04 N.S. 

Although there was a slight gain in student performance on tests 
of concept knowledge, this gain was not significant statistically. 
Anecdotal comments indicated that tie attribute frames used for 
categorization helped students develop rules for categorization. 
However, the categorizing process still presented students with 
di f ficulty . 

12 . Extrapolation 

Extrapolation was assessed using teacher observation data and 
teacher made tests. Observation data were supplied by 21 teachers. 
Seventeen reported that the technique increased students' abilities to 
see relationships between seemingly nrelated sets of information. 

Ten teachers reported test dat a. The results of two- tailed , 
t-test for dependent groups are reported below: 

Pre- test Post- test 

Mean Mean Difference d f T-value Sig 

.81 .95 .14 65 5.01 .001 

The analyses indicated a significant increase in students' 
abilit ies to extrapolate inform at ion. 
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Again teachers reported that pattern recognition was key to the 
extrapolation process. Once students recognized the pattern of 
information, they then had a framework which thev could use to identify 
similar information from a different context. 



13 . Analogical Reasoning 

Analogical reasoning was assessed using teacher observation and 
teacher made tests. Thirt y-seven teachers reported anecdotal dat a. Of 
those, 26 stated that direct teaching of the technique for analogical 
reasoning improved students 1 abilities to solve analogy problems 

Ten teachers reported pre-post test dat a. A two-t ai led t-test for 
dependent groups showed significant results at the .001 level 

Pre-test Post-test 

Mean Mean Difference df T-value Sig 

.86 .94 .08 91 7.35 .001 



As reported by teachers the most power lu! component of the 
analogical reasoning process was direct instruction in the semantic 
relationships found in analogy problems. Once students were dware of 
these relationships their ability to solve analogy problems increased. 



14 . Evaluation of Evidence 

Both teacher observation and teacher made test data were used to 
assess the skill, evaluation of evidence. Twenty-two teachers reported 
anecdotal data. Seventeen stated that they observed an increase in 
students' abilities to evaluate the logic of a claim. However, a 
two-tailed t-test for dependent groups on the test data reported by 
three teacherr did not show significant results. 

Pre-test Post-test 

Mean Mean Difference df T-value Sig 

.79 .81 .03 19 .79 N^S 



As an explanat ion of the discrepant results between anecdot al and 
classroom test data, teachers reported a low level of confidence in the 
validity of their tests. That is, they expressed doubt that their 
teacher mad'* tests actually measured student ability to evaluate 
evidence while at the same time expressing more confidence in their 
informal observations . 
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15 . Evaluation of Value 

Evaluation of value was assessed using teacher observation and 

student self-report data. Twenty five teachers reported anecdotal 
data. Of t/iose, 21 stated that the evaluation of value process 
increased students 1 aailities to identify the assumptions underlying 
their assignment of value and to identify different points of view 
which might generate different value judgments. 

Seven students were int erviewed about the effects of the 
evaluation of value process. All seven students indicated that the 
process provided them with a new level of insight into what attitudes 
and values are and how they are formed. 



16 . Elaboration 

Data on elaboration were collected via teacher observation only. 
Forty-one reachers reported. Of these, 37 stated that the elaboration 
process increased students' abilities to identify unstated 
characteristics, causes and intentions. That is, the elaboration 
process provided a framework which allowed students to identify implied 
information. Apparently this information was available to students but 
not easily accessed. The frameworks provided by the elaboration 
process apparently increased access. 



17 . Problem Solving 

Problem solving was assessed using teacher observation and student 
interviews. Eighteen teachers provided observational data. Twelve 
indicated that the problem solving techniques improved students' 
abilities to solve specific content related problems. Self-report data 
were collected from five students, all of whom indicated that the 
problem solving techniques presented them with a systematic way to 
approach and solve problems. 

Despite the generally positive results for the problem solving 
process many teachers reported that the problem solving heuristics 
presented to students were too general in nature. This implies a need 
for more specific problem solving heuristics for different problem 
types . 
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18 . Invention 

Invention was assessed via teacher observation »only. Twelve 
teachers reported. Of those, nine stated that the invention process 
significantly improved students ' abilities to conceive of, develop and 
carry through to completion projects involving the creation of new 
products. These teachers reported that the incubation and discovery 
phase of the invention process was the most useful component. It 
provided students with a method for translating imagery information 
into a linguistic form. 
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DISCUSSION 



All eighteen of the skills involved in the thinking skills program 
produced measurable effects in student behavior to one degree or 
another. This is n:>t surprising in light of the fact that the thinking 
skills roodel went through a set of screening phases described 
previously. One might say that the model which was formally tested was 
a fourth generation model. It would be expected, then, that those 
skills remaining after such a filtering process would produce positive 



Given the limitations to the evaluation design discussed 
previously these findings can not be considered stable. That is, given 
the threats to internal and external validity inherent in the 
evaluation design we cannot conclude that the measured effects for each 
skill area are a funct ion of the training program. However , given the 
st ate of the art of thinking skills instruction, the findings for the 
McREL program indicate that it compares well with other programs. More 
specifically in the major review of current thinking skill programs 
compiled by Segal, Chipman and Glaser (1983), it was reported that many 
of the current programs which are in wide use (soiae for over ten years) 
do not have a strong research base or a strong evaluation data base. 
In many cases, the McREL program actually has evaluation data that is 
more comprehensive and which is derived from a sounder evaluation 
design even though it has been in use a relatively short period of 
time. Current ly. controlled experiments are being conducted on each 
component of the McREL program to further refine the model and more 
specifically identify the effects produced Dy each component of the 
model . 



effects . 
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